1. Technical Field
The present disclosure relates to a motion navigation device for use with computer or other electronic devices employing graphical displays.
2. Description
As a conventional optical mouse moves along a surface, such as the surface of a desktop, it computes and provides to an associated computer its translational motion vectors (i.e. displacement components Δx, Δy, or velocity components if needed) relative to that surface. Using those translational motion vectors, the computer or other display-capable device can maneuver a cursor or other objects that are graphically displayed. However, the conventional optical mouse only provides two-dimensions (2D) of translational motion that the computer can implement. A third motional dimension, corresponding to rotational motions of the mouse, could produce a rotation of the object on the display that is directly correlated to those rotations. However, this third dimension of mouse motion is not provided to the computer, so an object on the display screen cannot easily or directly be made to rotate along with the corresponding rotation of the mouse.
In modern display-capable electronic devices, such a restricted motional functionality cannot satisfy the needs of many consumers. An increasing number of application programs require a 3D (three-dimensional) perspective when displaying and manipulating graphically generated objects. For such applications, the motion of the displayed object may require both translational and rotational components. For example, the computer graphics program AUTOCAD™ is used by many engineers and designers to render the drawings of mechanical parts with three-dimensional realism. As the AUTOCAD™ program gradually evolves into one with a full 3D capability, pipes, valves, and many similar realistically rendered mechanical parts will need to be rotated on the displaying device in a quick and accurate fashion, in response to the operator's instructions. Unfortunately, a “perspective adjustment” (i.e. changing the viewing angle of the object) cannot be done directly using the conventional optical mouse because the conventional optical mouse has only 2D (two-dimensional) functionality. Even though the operator may rotate the 2D mouse on its surface, the results of that rotation have no effective way of being implemented by the computer.
In the computer vision industry, which deals with the graphical display of the motion of a moving object, a vast amount of effort has been put into motion analysis using video/graphical rendering devices. In this area, the typical motion analysis tasks can be roughly characterized by, for example, 2D-2D, 2D-3D, and 3D-3D, depending on the correspondences between the device actually sensing the motion of a moving object (e.g. a CMOS image sensor) and the rendering device on which the moving object is graphically shown (e.g. a flat panel display). However, in contrast to the improvement in rendering devices, there have been fewer development efforts being applied to cursor maneuvering on a display screen. Such an improvement is an object of this disclosure.
To better understand the shortcomings of the prior art and for purposes of making comparisons with the present approach, we begin with Equ. 1 for the basic motion of an object under translations and rotations. For an object located at point p, having coordinates x, and y, which can be represented as a column vector.
      [                            x                                      y                      ]    ,its final position after going through a general maneuvering action can be determined by the translational displacement (denoted by the letter d in Equ. (1)) and rotational displacement (the rotational operator being denoted by letter M).
                              Mp          +          d                =                                            [                                                                                                                  S                        x                                            ⁢                      cos                      ⁢                                                                                          ⁢                                              θ                        x                                                                                                                                                -                                                  S                          y                                                                    ⁢                      sin                      ⁢                                                                                          ⁢                                              θ                        y                                                                                                                                                                                S                        x                                            ⁢                      sin                      ⁢                                                                                          ⁢                                              θ                        x                                                                                                                                                S                        y                                            ⁢                      cos                      ⁢                                                                                          ⁢                                              θ                        y                                                                                                        ]                        ⁡                          [                                                                    x                                                                                        y                                                              ]                                +                      [                                                                                d                    x                                                                                                                    d                    y                                                                        ]                                              (        1        )            
Here
         [                                        d            x                                                            d            y                                ]  denotes the translational displacement, M operating on p denotes the rotational displacement around the x and y axes (i.e., θx and θy), and S is the scaling factor along an axis (e.g. x or y).
The conventional optical mouse does not implement a parameter such as θx and θy (i.e. a rotation angle about an axis), in a way that can be easily calculated and used in the conventional art. Specifically, the motion vector communicated by the conventional optical mouse to the computer is only a translational motion vector. Limited by such a situation, as Equ. (1) reveals, it is only the quantity d, rather than M (i.e. M=0) that can be manipulated by the conventional mouse. When the situation requires a rotational movement for said object, M must be calculated by other means. For example, when an operator desires to rotate an object in a fully 3D manner, e.g. as in spinning a moving ball, the 2D motion vector provided by the conventional mouse must be continually recalculated by the computer so that a new pair of rotational angles θx and θy for the spinning ball is derived at each of the various points along its path. This procedure is not as simple as might be imagined and is implemented in two successive steps at each point where a rotation about an axis is desired. First, the computer program receiving the conventional mouse input must be instructed to stop accepting and treating the motion vectors sent by the conventional optical mouse as being translational movement. This is usually done by clicking a special button on the mouse. Second, the computer is instructed to designate an axis of rotation for the object of interest, so that the computer can then calculate the rotational vector about that axis using the translational motion vector (Δx, Δy) it has already received. Of course, the relative distance between the axis and the cursor's position on the display has vital influences on the angles, θx and θy, of rotation about that axis. Even for equal values, Δx, Δy, of the translational displacement, the calculated rotational angles θx and θy can be different if different pivots are selected. One must also bear in mind that pivots themselves are usually not displayed on the displaying device. Thus, the operator may easily ignore their existence. As the display program continues to operate, the pivot may change its position from time to time. For example, when the operator moves a mouse cursor over a crane machine graphically displayed on a 3D displaying device, the pivot point selected for rotational motion of the crane arm can be automatically changed by the computer in following the maneuvering act of the mouse cursor. Whenever there is a change in the pivot selection, the rotational motion vector derived using the same 2D movement of the mouse cursor is also changed. Thus, the operator can be easily confused by the ever changing results of shifting between translation and rotation when using the conventional optical mouse.
The increasing popularity of role-playing games (and applications of animated graphics generally) furnishes us with yet another example of the inadequacies of the 2D optical mouse in a 3D world. Consider the following imagined video game program sequence based on possible Disney (or similar) characters. In this imagined scenario Peter Pan is battling Captain Hook. Captain Hook is taller, and has a sword much longer than Peter Pan's dagger. To fight Captain Hook, Peter Pan must use his dagger with cleverness and agility. When Captain Hook's sword thrusts in, Peter Pan must swing his dagger and hop to avoid the sword, as the operator would undoubtedly have instructed him to do. To swing the dagger, the operator must give a rotational instruction. To hop, the operator must give the translational motion instructions. Unfortunately, the Peter Pan object created by today's computer game cannot easily swing his dagger from one angle to another following the mouse's instruction sent by the operator. Nor can Peter Pan easily fly around Captain Hook in a 3D arc. These long anticipated actions and movements of Peter Pan exist in every child's mind, but they are literally sacrificed by the technological shortcomings of a conventional optical mouse. This condition of motional limitation has been transmitted to all computer aided graphic arts. That is, almost all animated creatures in today's electronic graphic arts (e.g. Pixar's motion pictures) act robotically, lacking the delicate, gestures that can be produced by human arms and wrists. We will now present a schematic mathematical and pictorial example illustrating the shortcomings of the conventional art approach in transferring the motion of a conventional art mouse to a computer display.
FIG. 1 schematically depicts the successive positions of three exemplary targeted objects (105, 106, 107) located on a desktop over which an optical mouse might navigate. For simplicity, we assume they happen to lie initially on the circumference of a circle 110 with center 108 and that this is how they appear in an image sensor of a conventional art optical mouse at initial time t0. At a subsequent time t1, they will still lie on a circle, but the circle will now be labeled 111 and have center 109. This circle might actually be denoted a “pseudo-circle,” because it is not a physical structure on the desktop as are the targeted objects; the pseudo-circle will serve as a framework for the locations of the physical objects so that we can analyze their motion both algebraically and geometrically. For simplicity, however, the pseudo-circle will hereinafter simply be called a circle, unless we wish to emphasize its lack of physical properties. In addition, what we are now calling a “geometrical” analysis will later be denoted a topological analysis to allow for greater generality.
The image frame 101 formed by the mouse sensor is schematically represented as an 8×10 rectangular checkerboard pattern of pixels, using numbers vertically and letters horizontally as pixel labels.
At time t0, the three objects, shown as dark circles 105(t0), 106(t0), and 107(t0), are located, respectively, at pixels (C, 2), (H, 4), and (D, 7) in the image frame 101. At later time t1, they will be located at new positions 105(t1), 106(t1) and 107(t1), assumed to lie in the same image frame. The motion of the mouse producing this positional change of the three targetable objects in its image frame is assumed to be an arbitrary combination of a translation, T, of the circle 110 to become circle 111, and moving its center from 108 to 109, combined with a rotation, R, clockwise about the final circle center 109. We, therefore, describe the corresponding total displacement vectors of the three objects (or any one of them) within the image frame as being composed of both a translational displacement vector 104 (here, an arrow from the center 108 of the circle 110 to 109, the center of circle 111) and a rotational displacement vector 112 (here shown as the tangential displacement vector of 105(t0) on the circle 110).
At time t0, objects 105(t0), 106(t0), and 107(t0) are positioned on circle 110 whose geometrical center is labeled 108 (small dark dot). When the time has elapsed to t1, the three objects have moved to new locations (still assumed to be within the same image frame 101) which now lie on the circumference of circle 111, which has a center 109 (small dashed dot). Note that 108, which denotes the geometrical center of circle 110, is moved to a new position, as the center of circle 111, and is now denoted as 109. The vector 104, drawn as an arrow from 108 to 109, is the displacement vector of the circle center. The vector 112, also drawn as an arrow, describes the angular displacement of 105(t0), relative to the image frame. Note we treat this vector as being tangential to the circle and, for small rotations, it would have a length proportional to the product of the angular rotation of the circle (expressed in radians) and the radius of the circle.
Comparing the pixel addresses of the two circle centers 108 and 109, one observes that this “pseudo-object” (pseudo, because, like the circle itself, its center is not visible to the sensor) only moves from (E, 4.5) to (F, 4.5), there is no net motion in the vertical Y axis of the image frame. The two circle centers, 108 and 109, for the group entity formed by the location of the three objects on the circumference of a circle, remain on the horizontal line denoted 4.5. Prior art methods would have traced the motion of the individual objects (105), (106) and (107), from t1 to t1 and would have formed no concept of the circle 110 (or 111) as defining a group entity.
To analyze prior art's approach which fixes on the three real objects and not the pseudo-circle, we can decompose the net displacement of object 105(t0) into a translational motion vector and a rotational one, which results in the following description of its subsequent movement:    (1) Object 105 (t0) first moves from its original pixel location (C, 2) by the translational vector 104, which subsequently takes it to an intermediate position, which we will call the “translated” position 105(Trans).    (2) The intermediate translated object, 105(Trans), then moves to the next location in accord with rotational, R, motion vector 112, which takes it to the final position at (E, 1), denoted as the rotated position 105(Rot).    (3) The pixel address of the final object 105(Rot), is (E, 1). Comparing this final position to the initial position (C, 2) of object 105(t0), the object has moved by a net displacement vector with components determined as the difference in its pixel coordinates: ((E, 1)−(C, 2))=(2, −1), where we assign numerical values to the E and C positions.
Using the same technique on objects 106(t0) and 107(t0), one will find the net displacement vector of 106(t0) is (1, −1), and that of 107(t0) as (0, −1), respectively.
If the role played by rotational motions in this displacement process is not clearly understood, the discrepancies between the displacement vectors of 105(t0), 106(t0), and 107(t0) (i.e. (2, −1), (1, −1), and (0, −1) can be quite mysterious to the device operator. The device operator in the prior art would see no easy method to even recognize the role of rotations in this net motion, let alone to use them in a systematic way. It is simpler to make the effects of rotations “go away,” so that all motions of the prior art mouse are made to look like pure translations and are treated as such. In order to “bandage” the problem of unusable and unused rotations, the prior art may:                (A) Pick only objects of interest that lie on a very small circle (i.e. make the radius of circle 110 very small, so that the rotational motion vector (whose magnitude depends on the radius) can be ignored. This has a drawback—when the objects are congested in a narrow space, they may easily be merged with the one another as a single object body in the image frame. This problem will be exacerbated when the morphology of the targeted desktop surface is very flat. In fact it has become a common experience to almost all users that a very smooth or white surface may stall the conventional optical mouse easily (i.e. spurious spots are not differentiated from the targeted spots in the image frame).        (B) Use very high frame rate (e.g. >1000 frames/sec), so that the rotational motion vector between two consecutive image frames cannot be detected easily (i.e., the angular displacement is very small). The drawback of this tactic is that it leads to rapidly increasing power consumption.        (C) Use a coherent light source (e.g. laser) or dark field technique (this has more to do with the lens and aperture design) to differentiate the targeted objects from the background scene. These techniques are the facilitative means of tactic (A). Coherent light helps the prior art pick out objects from even smaller area (i.e. the radius of circle 102 is even smaller). A laser light source, as can be expected, is power consuming.        
As one may easily conclude from the necessity of using any of the above tactics, the prior art has not developed a fast and robust means to derive and use the actual rotational motion vectors. Although the prior art does teach various approaches to address these issues, such as are to be found in Geoffrey Lee (U.S. Pat. No. 6,392,632), Wen-Chieh Geoffrey Lee (US Pat. Appl. 2013/002555), where both of the previous prior arts are fully incorporated herein by reference. Also, see Hattori et al. (U.S. Pat. No. 8,279,279), Unsworth (US Pat. Appl. 2010/0036393), Fouquet et al. (US Pat. Appl. 2010/0001950), Dalal (US Pat. Appl. 2011/0304541), Zahnert et al. (US Pat. Appl. 2011/0234815) and Krah et al. (U.S. Pat. No. 8,077,147). However, none of these approaches will provide the present device and teach its operation as will be described below.
We now summarize the foregoing discussion. The prior art computer mouse is basically a maneuvering device for a cursor or other graphically generated object displayed on a computer screen or similar display screen. It provides that maneuverability by determining its own motion relative to a surface that contains identifiable and traceable objects. That motion is transmitted to the computer where it is converted to on-screen motion of a graphically generated object. Although the motion of the prior art mouse contains rotational as well as translational elements, there is no practical way to identify or compute the rotational elements, so their existence is in effect ignored, and the mouse sends data to the computer indicating all of its motions as being translational. In order that this intentional fiction shall not create serious divergences between actual mouse motion and on-screen motion, the prior art mouse must be operated in such a way that the discrepancies are minimized and that effects that to not correspond well to translational motion are treated as noise.
When considering the positional accuracy of a cursor maneuvering device, the prior art often refers to two parameters—the SQUAL (Surface QUALity), a surface parameter associated with the numbers of pixels in an image frame being identified as a traceable object, and the dpi (dots per inch), a mouse parameter, giving the number of dots per square inch of the surface that are resolvable as pixels in an image frame.
The parameter SQUAL in effect is used to denote the quality (i.e. presence of identifiable objects) of the surface upon which the mouse unit is positioned. For example, when an optical mouse is placed on a flat surface, the SQUAL value is low, while on a rough surface it is high. A surface with low SQUAL value (e.g. a glossy, flat, or white surface) will provide a low dpi value; by the same token, a surface with high SQUAL (e.g. a rough surface) will provide higher dpi. The resolution of the mouse, on the other hand, is provided by its dpi. Of course, the motion vectors generated by a mouse unit with high dpi will provide higher positional accuracy for a cursor or for the object displayed in the graphical rendering device. A question that still remains unanswered is how a motion vector generated by motion relative to a surface with a specific SQUAL index is to be correlated to the positional accuracy of a cursor maneuvering device.